In [1]:
import pandas as pd
import numpy as np
import plotly.express as plx
from plotly.subplots import make_subplots
import plotly.graph_objects as go

We perform energy analysis using 12 different building shapes simulated in Ecotect. The buildings differ with respect to the glazing area, the glazing area distribution, and the orientation, amongst other parameters.¶

We simulate various settings as functions of the afore-mentioned characteristics to obtain 768 building shapes.¶

The dataset comprises 768 samples and 8 features, aiming to predict two real valued responses.¶

It can also be used as a multi-class classification problem if the response is rounded to the nearest integer.¶

The dataset contains eight attributes (or features, denoted by X1...X8) and two responses (or outcomes, denoted by y1 and y2). The aim is to use the eight features to predict each of the two responses.¶

X1 Relative Compactness¶

X2 Surface Area¶

X3 Wall Area¶

X4 Roof Area¶

X5 Overall Height¶

X6 Orientation¶

X7 Glazing Area¶

X8 Glazing Area Distribution¶

y1 Heating Load¶

y2 Cooling Load¶

In [2]:
df = pd.DataFrame(pd.read_excel('C:\\Users\\harip\\INEURON_PROJECTS\\Energy Efficiency\\energy+efficiency\\ENB2012_data.xlsx'))
df
Out[2]:
X1 X2 X3 X4 X5 X6 X7 X8 Y1 Y2
0 0.98 514.5 294.0 110.25 7.0 2 0.0 0 15.55 21.33
1 0.98 514.5 294.0 110.25 7.0 3 0.0 0 15.55 21.33
2 0.98 514.5 294.0 110.25 7.0 4 0.0 0 15.55 21.33
3 0.98 514.5 294.0 110.25 7.0 5 0.0 0 15.55 21.33
4 0.90 563.5 318.5 122.50 7.0 2 0.0 0 20.84 28.28
... ... ... ... ... ... ... ... ... ... ...
763 0.64 784.0 343.0 220.50 3.5 5 0.4 5 17.88 21.40
764 0.62 808.5 367.5 220.50 3.5 2 0.4 5 16.54 16.88
765 0.62 808.5 367.5 220.50 3.5 3 0.4 5 16.44 17.11
766 0.62 808.5 367.5 220.50 3.5 4 0.4 5 16.48 16.61
767 0.62 808.5 367.5 220.50 3.5 5 0.4 5 16.64 16.03

768 rows × 10 columns

In [3]:
df.isnull().sum()
Out[3]:
X1    0
X2    0
X3    0
X4    0
X5    0
X6    0
X7    0
X8    0
Y1    0
Y2    0
dtype: int64

we already know that, there is no missing values in the dataset from https://archive.ics.uci.edu/dataset/242/energy+efficiency¶

In [4]:
df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 768 entries, 0 to 767
Data columns (total 10 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   X1      768 non-null    float64
 1   X2      768 non-null    float64
 2   X3      768 non-null    float64
 3   X4      768 non-null    float64
 4   X5      768 non-null    float64
 5   X6      768 non-null    int64  
 6   X7      768 non-null    float64
 7   X8      768 non-null    int64  
 8   Y1      768 non-null    float64
 9   Y2      768 non-null    float64
dtypes: float64(8), int64(2)
memory usage: 60.1 KB
In [5]:
df.describe()
Out[5]:
X1 X2 X3 X4 X5 X6 X7 X8 Y1 Y2
count 768.000000 768.000000 768.000000 768.000000 768.00000 768.000000 768.000000 768.00000 768.000000 768.000000
mean 0.764167 671.708333 318.500000 176.604167 5.25000 3.500000 0.234375 2.81250 22.307195 24.587760
std 0.105777 88.086116 43.626481 45.165950 1.75114 1.118763 0.133221 1.55096 10.090204 9.513306
min 0.620000 514.500000 245.000000 110.250000 3.50000 2.000000 0.000000 0.00000 6.010000 10.900000
25% 0.682500 606.375000 294.000000 140.875000 3.50000 2.750000 0.100000 1.75000 12.992500 15.620000
50% 0.750000 673.750000 318.500000 183.750000 5.25000 3.500000 0.250000 3.00000 18.950000 22.080000
75% 0.830000 741.125000 343.000000 220.500000 7.00000 4.250000 0.400000 4.00000 31.667500 33.132500
max 0.980000 808.500000 416.500000 220.500000 7.00000 5.000000 0.400000 5.00000 43.100000 48.030000

In this dataset, there are two output variables and 8 input variables.¶

The datatype of output variables (Y1 & Y2) are float.¶

The datatype of 6 input variables(X1,X2,X3,X4,X5,X7) are float and remaining 2 variables(X6,X8) are integer.¶

Exploratory Data Analysis¶

In [6]:
plx.imshow(df.corr(),height=750,width=750,text_auto=True)

Insights:¶

From the above correlation heatmap we know that,¶

1. Y1 and Y2 are highly positively correlated to each other.¶

2. X5(Overall Height of the building) is highly positively correlated with both the output variables (Y1 & Y2). What literally means is, if overall height of the building increases then heating and cooling load is also increased.¶

3. X4(Area of the Roof of building) is highly negatively correlated with both the output variable (Y1 & Y2). It means that, if roof area of the building increases then heating load and cooling load is decreased.¶

4. X1 and X3 (Relative Compactness and Wall Area) having moderate correlation with output variables(Y1 & Y2).¶

5. X2 (Surface Area) having moderate negative correlation with output variables(Y1 & Y2).¶

6. X3(Wall Area) variable is only important for output variables(Y1 & Y2), not for other 7 variables.¶

7. X1,X2,X4 and X5(Relative Compactness, Surface Area, Roof Area and Overall Height) are highly correlated with each other.¶

8. X7(Glazing Area) having around 0.2 correlation with both the ouput variables(Y1 & Y2) and X8 variable also, but not important for other 6 variables.¶

9. X6 (Orientation) is not important for any of the feature including the output variables (Y1 & Y2). What is Orientation and Why this is not important? Here, Orientation is nothing but how the building is positioned. We got explanation from google -----> "Orientation is how a building is positioned in relation to the sun's paths in different seasons, as well as to prevailing wind patterns. In passive design, it is also about how living and sleeping areas are designed and positioned, either to take advantage of the sun and wind, or be protected from their effects". Here, the data is in number format 2,3,4,5. It has some meaning, but we don't know what literally is. From the insights we got, we clearly know that, Y1 and Y2 is not based on the Orientation of the building and it is not important.¶

10. X8 is not important for the ouput variables(Y1 & Y2), but having corrleation of 0.2 with variable X7.¶

11. X1 & X2 are highly negatively correlated with each other.¶

In [7]:
df.corr()['X6']
Out[7]:
X1    4.678592e-17
X2   -3.459372e-17
X3   -2.429499e-17
X4   -5.830058e-17
X5    4.492205e-17
X6    1.000000e+00
X7   -9.406007e-16
X8   -2.549352e-16
Y1   -2.586763e-03
Y2    1.428960e-02
Name: X6, dtype: float64
In [8]:
df['X6'].value_counts()
Out[8]:
2    192
3    192
4    192
5    192
Name: X6, dtype: int64
In [9]:
df['X8'].value_counts()
Out[9]:
1    144
2    144
3    144
4    144
5    144
0     48
Name: X8, dtype: int64
In [10]:
plx.box(x = df['X8'],y=df['Y1'],color=df['X8'])
In [11]:
plx.box(x = df['X8'],y=df['Y2'],color=df['X8'])

From the above two graph, we know that 0 belongs to one group and rest belongs to one group.¶

Dimensionality Reduction¶

In [12]:
plx.imshow(df.corr(),height=750,width=750,text_auto=True)

From the above heatmap, we know that X6(Orientation) is not important for any of the variables including output variables(Y1 & Y2). So, we can remove X6(Orientation) from the context.¶

In [13]:
df.drop(['X6'],axis=1,inplace=True)
In [14]:
df
Out[14]:
X1 X2 X3 X4 X5 X7 X8 Y1 Y2
0 0.98 514.5 294.0 110.25 7.0 0.0 0 15.55 21.33
1 0.98 514.5 294.0 110.25 7.0 0.0 0 15.55 21.33
2 0.98 514.5 294.0 110.25 7.0 0.0 0 15.55 21.33
3 0.98 514.5 294.0 110.25 7.0 0.0 0 15.55 21.33
4 0.90 563.5 318.5 122.50 7.0 0.0 0 20.84 28.28
... ... ... ... ... ... ... ... ... ...
763 0.64 784.0 343.0 220.50 3.5 0.4 5 17.88 21.40
764 0.62 808.5 367.5 220.50 3.5 0.4 5 16.54 16.88
765 0.62 808.5 367.5 220.50 3.5 0.4 5 16.44 17.11
766 0.62 808.5 367.5 220.50 3.5 0.4 5 16.48 16.61
767 0.62 808.5 367.5 220.50 3.5 0.4 5 16.64 16.03

768 rows × 9 columns

From the above heatmap, we know that X1 and X2 are highly negatively correlated with each other. We have remove the feature which having less correlation with respect to output variables.¶

X1 having around 0.63 correlation on output variables Y1 & Y2.¶

X2 having around -0.67 correlation on output variables Y1 & Y2.¶

Here, we have to remove X1(Relative Compactness)¶

In [15]:
df.drop(['X1'],axis=1,inplace=True)
In [16]:
df
Out[16]:
X2 X3 X4 X5 X7 X8 Y1 Y2
0 514.5 294.0 110.25 7.0 0.0 0 15.55 21.33
1 514.5 294.0 110.25 7.0 0.0 0 15.55 21.33
2 514.5 294.0 110.25 7.0 0.0 0 15.55 21.33
3 514.5 294.0 110.25 7.0 0.0 0 15.55 21.33
4 563.5 318.5 122.50 7.0 0.0 0 20.84 28.28
... ... ... ... ... ... ... ... ...
763 784.0 343.0 220.50 3.5 0.4 5 17.88 21.40
764 808.5 367.5 220.50 3.5 0.4 5 16.54 16.88
765 808.5 367.5 220.50 3.5 0.4 5 16.44 17.11
766 808.5 367.5 220.50 3.5 0.4 5 16.48 16.61
767 808.5 367.5 220.50 3.5 0.4 5 16.64 16.03

768 rows × 8 columns

From the above heatmap, we know that X4 and X5 are highly negatively correlated with each other. We have remove the feature which having less correlation with respect to output variables.¶

In [17]:
df.drop(['X4'],axis=1,inplace=True)
In [18]:
df
Out[18]:
X2 X3 X5 X7 X8 Y1 Y2
0 514.5 294.0 7.0 0.0 0 15.55 21.33
1 514.5 294.0 7.0 0.0 0 15.55 21.33
2 514.5 294.0 7.0 0.0 0 15.55 21.33
3 514.5 294.0 7.0 0.0 0 15.55 21.33
4 563.5 318.5 7.0 0.0 0 20.84 28.28
... ... ... ... ... ... ... ...
763 784.0 343.0 3.5 0.4 5 17.88 21.40
764 808.5 367.5 3.5 0.4 5 16.54 16.88
765 808.5 367.5 3.5 0.4 5 16.44 17.11
766 808.5 367.5 3.5 0.4 5 16.48 16.61
767 808.5 367.5 3.5 0.4 5 16.64 16.03

768 rows × 7 columns

we can change the value of X8. 0 is represented as 0 and rest of them represented as 1.¶

In [19]:
df.loc[(df['X8']>0), 'X8'] = 1
df
Out[19]:
X2 X3 X5 X7 X8 Y1 Y2
0 514.5 294.0 7.0 0.0 0 15.55 21.33
1 514.5 294.0 7.0 0.0 0 15.55 21.33
2 514.5 294.0 7.0 0.0 0 15.55 21.33
3 514.5 294.0 7.0 0.0 0 15.55 21.33
4 563.5 318.5 7.0 0.0 0 20.84 28.28
... ... ... ... ... ... ... ...
763 784.0 343.0 3.5 0.4 1 17.88 21.40
764 808.5 367.5 3.5 0.4 1 16.54 16.88
765 808.5 367.5 3.5 0.4 1 16.44 17.11
766 808.5 367.5 3.5 0.4 1 16.48 16.61
767 808.5 367.5 3.5 0.4 1 16.64 16.03

768 rows × 7 columns

Now, the values of X8 gets changed.¶

In [20]:
plx.imshow(df.corr(),height=750,width=750,text_auto=True)

Insight¶

1. After change in X8 feature, the correlation between X7 and X8 is increased and also with output variable Y1 & Y2.¶

In [21]:
df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 768 entries, 0 to 767
Data columns (total 7 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   X2      768 non-null    float64
 1   X3      768 non-null    float64
 2   X5      768 non-null    float64
 3   X7      768 non-null    float64
 4   X8      768 non-null    int64  
 5   Y1      768 non-null    float64
 6   Y2      768 non-null    float64
dtypes: float64(6), int64(1)
memory usage: 42.1 KB
In [22]:
plx.box(x = df['X8'],y=df['Y1'],color=df['X8'])
In [23]:
plx.box(x = df['X8'],y=df['Y1'],color=df['X8'])
In [24]:
len(df)
Out[24]:
768
In [25]:
df.drop_duplicates(inplace=True)
df
Out[25]:
X2 X3 X5 X7 X8 Y1 Y2
0 514.5 294.0 7.0 0.0 0 15.55 21.33
4 563.5 318.5 7.0 0.0 0 20.84 28.28
5 563.5 318.5 7.0 0.0 0 21.46 25.38
6 563.5 318.5 7.0 0.0 0 20.71 25.16
7 563.5 318.5 7.0 0.0 0 19.68 29.60
... ... ... ... ... ... ... ...
763 784.0 343.0 3.5 0.4 1 17.88 21.40
764 808.5 367.5 3.5 0.4 1 16.54 16.88
765 808.5 367.5 3.5 0.4 1 16.44 17.11
766 808.5 367.5 3.5 0.4 1 16.48 16.61
767 808.5 367.5 3.5 0.4 1 16.64 16.03

756 rows × 7 columns

We found that 12 duplicates were removed from the dataframe.¶

Now, Everything is good in dataset. We can move forward to the model development.¶

Model Development¶

In [26]:
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.linear_model import SGDRegressor
from sklearn.linear_model import ElasticNet
from sklearn.ensemble import AdaBoostRegressor
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.ensemble import RandomForestRegressor
from sklearn.svm import SVR
from xgboost import XGBRegressor
from catboost import CatBoostRegressor
from sklearn.multioutput import MultiOutputRegressor
from sklearn.metrics import r2_score
from tqdm import tqdm
In [27]:
X = df.drop(['Y1','Y2'],axis=1)
Y = df.drop(['X2','X3','X5','X7','X8'],axis=1)
lr_trn_score,rfr_trn_score,sgd_trn_score,en_trn_score,abr_trn_score,gbr_trn_score,svr_trn_score,xgb_trn_score,cbr_trn_score = [],[],[],[],[],[],[],[],[]
lr_test_score,rfr_test_score,sgd_test_score,en_test_score,abr_test_score,gbr_test_score,svr_test_score,xgb_test_score,cbr_test_score = [],[],[],[],[],[],[],[],[]
for i in tqdm(range(1000)):
    x_train, x_test, y_train, y_test = train_test_split(X,Y,test_size=0.2)
    
    lr = LinearRegression().fit(x_train, y_train)
    pred = lr.predict(x_test)
    pred_trn = lr.predict(x_train)
    lr_test_score.append(r2_score(y_test, pred))
    lr_trn_score.append(r2_score(y_train, pred_trn))
    
    sgd = MultiOutputRegressor(SGDRegressor()).fit(x_train,y_train)
    pred = sgd.predict(x_test)
    pred_trn = sgd.predict(x_train)
    sgd_test_score.append(r2_score(y_test, pred))
    sgd_trn_score.append(r2_score(y_train, pred_trn))
    
    en = ElasticNet().fit(x_train,y_train)
    pred = en.predict(x_test)
    pred_trn = en.predict(x_train)
    en_test_score.append(r2_score(y_test, pred))
    en_trn_score.append(r2_score(y_train, pred_trn))
    
    abr = MultiOutputRegressor(AdaBoostRegressor()).fit(x_train,y_train)
    pred = abr.predict(x_test)
    pred_trn = abr.predict(x_train)
    abr_test_score.append(r2_score(y_test, pred))
    abr_trn_score.append(r2_score(y_train, pred_trn))
    
    gbr = MultiOutputRegressor(GradientBoostingRegressor()).fit(x_train,y_train)
    pred = gbr.predict(x_test)
    pred_trn = gbr.predict(x_train)
    gbr_test_score.append(r2_score(y_test, pred))
    gbr_trn_score.append(r2_score(y_train, pred_trn))
    
    svr = MultiOutputRegressor(SVR()).fit(x_train,y_train)
    pred = svr.predict(x_test)
    pred_trn = svr.predict(x_train)
    svr_test_score.append(r2_score(y_test, pred))
    svr_trn_score.append(r2_score(y_train, pred_trn))
    
    xgb = MultiOutputRegressor(XGBRegressor()).fit(x_train,y_train)
    pred = xgb.predict(x_test)
    pred_trn = xgb.predict(x_train)
    xgb_test_score.append(r2_score(y_test, pred))
    xgb_trn_score.append(r2_score(y_train, pred_trn))
    
    cbr = MultiOutputRegressor(CatBoostRegressor(verbose=0)).fit(x_train,y_train)
    pred = cbr.predict(x_test)
    pred_trn = cbr.predict(x_train)
    cbr_test_score.append(r2_score(y_test, pred))
    cbr_trn_score.append(r2_score(y_train, pred_trn))
    
    
    rfr = RandomForestRegressor().fit(x_train, y_train)
    pred = rfr.predict(x_test)
    pred_trn = lr.predict(x_train)
    rfr_test_score.append(r2_score(y_test, pred))
    rfr_trn_score.append(r2_score(y_train, pred_trn))

100%|██████████| 1000/1000 [25:04<00:00,  1.50s/it]

1. Linear Regression¶

In [28]:
fig = make_subplots(rows = 2, cols = 1)
fig.append_trace(go.Scatter(y = lr_test_score, name = 'Test Score'), row=1, col=1)
fig.append_trace(go.Scatter(y = lr_trn_score, name = 'Train Score'), row=2, col=1)
fig.update_layout(title = 'Train vs Test Score on Linear Regression')
fig.show()

2. SGDRegressor¶

In [29]:
fig = make_subplots(rows = 2, cols = 1)
fig.append_trace(go.Scatter(y = sgd_test_score, name = 'Test Score'), row=1, col=1)
fig.append_trace(go.Scatter(y = sgd_trn_score, name = 'Train Score'), row=2, col=1)
fig.update_layout(title = 'Train vs Test Score on SGDRegressor')
fig.show()

3. ElasticNet¶

In [30]:
fig = make_subplots(rows = 2, cols = 1)
fig.append_trace(go.Scatter(y = en_test_score, name = 'Test Score'), row=1, col=1)
fig.append_trace(go.Scatter(y = en_trn_score, name = 'Train Score'), row=2, col=1)
fig.update_layout(title = 'Train vs Test Score on ElasticNet Regression')
fig.show()

4. AdaBoostRegressor¶

In [31]:
fig = make_subplots(rows = 2, cols = 1)
fig.append_trace(go.Scatter(y = abr_test_score, name = 'Test Score'), row=1, col=1)
fig.append_trace(go.Scatter(y = abr_trn_score, name = 'Train Score'), row=2, col=1)
fig.update_layout(title = 'Train vs Test Score on AdaBoostRegressor')
fig.show()

5.GradientBoostingRegressor¶

In [32]:
fig = make_subplots(rows = 2, cols = 1)
fig.append_trace(go.Scatter(y = gbr_test_score, name = 'Test Score'), row=1, col=1)
fig.append_trace(go.Scatter(y = gbr_trn_score, name = 'Train Score'), row=2, col=1)
fig.update_layout(title = 'Train vs Test Score on GradientBoostingRegressor')
fig.show()

6. SVR¶

In [33]:
fig = make_subplots(rows = 2, cols = 1)
fig.append_trace(go.Scatter(y = svr_test_score, name = 'Test Score'), row=1, col=1)
fig.append_trace(go.Scatter(y = svr_trn_score, name = 'Train Score'), row=2, col=1)
fig.update_layout(title = 'Train vs Test Score on Support Vector Regressor')
fig.show()

7. XGBRegressor¶

In [34]:
fig = make_subplots(rows = 2, cols = 1)
fig.append_trace(go.Scatter(y = xgb_test_score, name = 'Test Score'), row=1, col=1)
fig.append_trace(go.Scatter(y = xgb_trn_score, name = 'Train Score'), row=2, col=1)
fig.update_layout(title = 'Train vs Test Score on XGBRegressor')
fig.show()

8. CatBoostRegressor¶

In [35]:
fig = make_subplots(rows = 2, cols = 1)
fig.append_trace(go.Scatter(y = cbr_test_score, name = 'Test Score'), row=1, col=1)
fig.append_trace(go.Scatter(y = cbr_trn_score, name = 'Train Score'), row=2, col=1)
fig.update_layout(title = 'Train vs Test Score on CatBoostRegressor')
fig.show()

9. RandomForestRegressor¶

In [36]:
fig = make_subplots(rows = 2, cols = 1)
fig.append_trace(go.Scatter(y = rfr_test_score, name = 'Test Score'), row=1, col=1)
fig.append_trace(go.Scatter(y = rfr_trn_score, name = 'Train Score'), row=2, col=1)
fig.update_layout(title = 'Train vs Test Score on RandomForestRegressor')
fig.show()

From the above visualization, we know that Boosting Algorithm predicts better than other algorithms. Both, train and test score(r2_score) is good is Boosting Algorithm around 0.985.¶

In [37]:
X = df.drop(['Y1','Y2'],axis=1)
Y = df.drop(['X2','X3','X5','X7','X8'],axis=1)
x_train, x_test, y_train, y_test = train_test_split(X,Y,test_size=0.2)
cbr = MultiOutputRegressor(CatBoostRegressor(verbose=0, n_estimators=10000,early_stopping_rounds=100)).fit(x_train,y_train)
pred = cbr.predict(x_test)
pred_trn = cbr.predict(x_train)
print(r2_score(y_train, pred_trn), r2_score(y_test, pred))

0.9846453823145591 0.9849155431050601
In [38]:
y1_pred,y2_pred = [],[]
for i in range(len(pred)):
    y1_pred.append(pred[i][0])
    y2_pred.append(pred[i][1])
In [39]:
def visulaize_performance_of_the_model(pred, y_test, modelname):
    # Plotting both line & scatter plot in same graph of predicted values to check the performance of the model in visualization.
    fig = go.Figure()
    fig.add_trace(go.Scatter(x=np.arange(0,50), y=np.arange(0,50),
                             mode='lines',
                             name='perfectline'))
    fig.add_trace(go.Scatter(x=pred, y=y_test,
                             mode='markers',
                             name='predictions'))
    fig.update_layout(
        title=f"Performance of {modelname} on Test data",
        xaxis_title="Predicted",
        yaxis_title="Actual",
        font=dict(
            family="Courier New, monospace",
            size=13,
            color="RebeccaPurple"
        )
    )
    fig.show()
In [40]:
visulaize_performance_of_the_model(y1_pred, y_test['Y1'], 'CatBoost regressor')
In [41]:
visulaize_performance_of_the_model(y2_pred, y_test['Y2'], 'CatBoost regressor')

From the above graph, we know that prediction on Y1 is too good. But on Y2, a bit poor when comapred to Y1. So, we decided to predict Y1 and Y2 individually with 2 models. Let's find the best algorithm for Y1 and Y2.¶

1. Prediction on Y1(Heating Load)¶

In [42]:
X = df.drop(['Y1','Y2'],axis=1)
Y = df['Y1']
lr_trn_score,rfr_trn_score,abr_trn_score,gbr_trn_score,xgb_trn_score,cbr_trn_score = [],[],[],[],[],[]
lr_test_score,rfr_test_score,abr_test_score,gbr_test_score,xgb_test_score,cbr_test_score = [],[],[],[],[],[]
for i in tqdm(range(1000)):
    x_train, x_test, y_train, y_test = train_test_split(X,Y,test_size=0.2)
    
    lr = LinearRegression().fit(x_train, y_train)
    pred = lr.predict(x_test)
    pred_trn = lr.predict(x_train)
    lr_test_score.append(r2_score(y_test, pred))
    lr_trn_score.append(r2_score(y_train, pred_trn))
    
    abr = AdaBoostRegressor().fit(x_train,y_train)
    pred = abr.predict(x_test)
    pred_trn = abr.predict(x_train)
    abr_test_score.append(r2_score(y_test, pred))
    abr_trn_score.append(r2_score(y_train, pred_trn))
    
    gbr = GradientBoostingRegressor().fit(x_train,y_train)
    pred = gbr.predict(x_test)
    pred_trn = gbr.predict(x_train)
    gbr_test_score.append(r2_score(y_test, pred))
    gbr_trn_score.append(r2_score(y_train, pred_trn))
     
    xgb = XGBRegressor().fit(x_train,y_train)
    pred = xgb.predict(x_test)
    pred_trn = xgb.predict(x_train)
    xgb_test_score.append(r2_score(y_test, pred))
    xgb_trn_score.append(r2_score(y_train, pred_trn))
    
    cbr = CatBoostRegressor(verbose=0).fit(x_train,y_train)
    pred = cbr.predict(x_test)
    pred_trn = cbr.predict(x_train)
    cbr_test_score.append(r2_score(y_test, pred))
    cbr_trn_score.append(r2_score(y_train, pred_trn))
    
    
    rfr = RandomForestRegressor().fit(x_train, y_train)
    pred = rfr.predict(x_test)
    pred_trn = lr.predict(x_train)
    rfr_test_score.append(r2_score(y_test, pred))
    rfr_trn_score.append(r2_score(y_train, pred_trn))

100%|██████████| 1000/1000 [12:30<00:00,  1.33it/s]

1. Linear Regression¶

In [43]:
fig = make_subplots(rows = 2, cols = 1)
fig.append_trace(go.Scatter(y = lr_test_score, name = 'Test Score'), row=1, col=1)
fig.append_trace(go.Scatter(y = lr_trn_score, name = 'Train Score'), row=2, col=1)
fig.update_layout(title = 'Train vs Test Score on Linear Regression')
fig.show()

2. AdaBoostRegressor¶

In [44]:
fig = make_subplots(rows = 2, cols = 1)
fig.append_trace(go.Scatter(y = abr_test_score, name = 'Test Score'), row=1, col=1)
fig.append_trace(go.Scatter(y = abr_trn_score, name = 'Train Score'), row=2, col=1)
fig.update_layout(title = 'Train vs Test Score on AdaBoostRegressor')
fig.show()

3. GradientBoostingRegressor¶

In [45]:
fig = make_subplots(rows = 2, cols = 1)
fig.append_trace(go.Scatter(y = gbr_test_score, name = 'Test Score'), row=1, col=1)
fig.append_trace(go.Scatter(y = gbr_trn_score, name = 'Train Score'), row=2, col=1)
fig.update_layout(title = 'Train vs Test Score on GradientBoostingRegressor')
fig.show()

4. XGBRegressor¶

In [46]:
fig = make_subplots(rows = 2, cols = 1)
fig.append_trace(go.Scatter(y = xgb_test_score, name = 'Test Score'), row=1, col=1)
fig.append_trace(go.Scatter(y = xgb_trn_score, name = 'Train Score'), row=2, col=1)
fig.update_layout(title = 'Train vs Test Score on XGBRegressor')
fig.show()

5. CatBoostRegressor¶

In [47]:
fig = make_subplots(rows = 2, cols = 1)
fig.append_trace(go.Scatter(y = cbr_test_score, name = 'Test Score'), row=1, col=1)
fig.append_trace(go.Scatter(y = cbr_trn_score, name = 'Train Score'), row=2, col=1)
fig.update_layout(title = 'Train vs Test Score on CatBoostRegressor')
fig.show()

6. RandomForestRegressor¶

In [48]:
fig = make_subplots(rows = 2, cols = 1)
fig.append_trace(go.Scatter(y = rfr_test_score, name = 'Test Score'), row=1, col=1)
fig.append_trace(go.Scatter(y = rfr_trn_score, name = 'Train Score'), row=2, col=1)
fig.update_layout(title = 'Train vs Test Score on RandomForestRegressor')
fig.show()

2. Prediction on Y2(Cooling Load)¶

In [49]:
X = df.drop(['Y1','Y2'],axis=1)
Y = df['Y2']
lr_trn_score,rfr_trn_score,abr_trn_score,gbr_trn_score,xgb_trn_score,cbr_trn_score = [],[],[],[],[],[]
lr_test_score,rfr_test_score,abr_test_score,gbr_test_score,xgb_test_score,cbr_test_score = [],[],[],[],[],[]
for i in tqdm(range(1000)):
    x_train, x_test, y_train, y_test = train_test_split(X,Y,test_size=0.2)
    
    lr = LinearRegression().fit(x_train, y_train)
    pred = lr.predict(x_test)
    pred_trn = lr.predict(x_train)
    lr_test_score.append(r2_score(y_test, pred))
    lr_trn_score.append(r2_score(y_train, pred_trn))
    
    abr = AdaBoostRegressor().fit(x_train,y_train)
    pred = abr.predict(x_test)
    pred_trn = abr.predict(x_train)
    abr_test_score.append(r2_score(y_test, pred))
    abr_trn_score.append(r2_score(y_train, pred_trn))
    
    gbr = GradientBoostingRegressor().fit(x_train,y_train)
    pred = gbr.predict(x_test)
    pred_trn = gbr.predict(x_train)
    gbr_test_score.append(r2_score(y_test, pred))
    gbr_trn_score.append(r2_score(y_train, pred_trn))
     
    xgb = XGBRegressor().fit(x_train,y_train)
    pred = xgb.predict(x_test)
    pred_trn = xgb.predict(x_train)
    xgb_test_score.append(r2_score(y_test, pred))
    xgb_trn_score.append(r2_score(y_train, pred_trn))
    
    cbr = CatBoostRegressor(verbose=0).fit(x_train,y_train)
    pred = cbr.predict(x_test)
    pred_trn = cbr.predict(x_train)
    cbr_test_score.append(r2_score(y_test, pred))
    cbr_trn_score.append(r2_score(y_train, pred_trn))
    
    
    rfr = RandomForestRegressor().fit(x_train, y_train)
    pred = rfr.predict(x_test)
    pred_trn = lr.predict(x_train)
    rfr_test_score.append(r2_score(y_test, pred))
    rfr_trn_score.append(r2_score(y_train, pred_trn))

100%|██████████| 1000/1000 [12:00<00:00,  1.39it/s]

1. Linear Regression¶

In [50]:
fig = make_subplots(rows = 2, cols = 1)
fig.append_trace(go.Scatter(y = lr_test_score, name = 'Test Score'), row=1, col=1)
fig.append_trace(go.Scatter(y = lr_trn_score, name = 'Train Score'), row=2, col=1)
fig.update_layout(title = 'Train vs Test Score on Linear Regression')
fig.show()

2. AdaBoostRegressor¶

In [51]:
fig = make_subplots(rows = 2, cols = 1)
fig.append_trace(go.Scatter(y = abr_test_score, name = 'Test Score'), row=1, col=1)
fig.append_trace(go.Scatter(y = abr_trn_score, name = 'Train Score'), row=2, col=1)
fig.update_layout(title = 'Train vs Test Score on AdaBoostRegressor')
fig.show()

3. GradientBoostingRegressor¶

In [52]:
fig = make_subplots(rows = 2, cols = 1)
fig.append_trace(go.Scatter(y = gbr_test_score, name = 'Test Score'), row=1, col=1)
fig.append_trace(go.Scatter(y = gbr_trn_score, name = 'Train Score'), row=2, col=1)
fig.update_layout(title = 'Train vs Test Score on GradientBoostingRegressor')
fig.show()

4. XGBRegressor¶

In [53]:
fig = make_subplots(rows = 2, cols = 1)
fig.append_trace(go.Scatter(y = xgb_test_score, name = 'Test Score'), row=1, col=1)
fig.append_trace(go.Scatter(y = xgb_trn_score, name = 'Train Score'), row=2, col=1)
fig.update_layout(title = 'Train vs Test Score on XGBRegressor')
fig.show()

5. CatBoostRegressor¶

In [54]:
fig = make_subplots(rows = 2, cols = 1)
fig.append_trace(go.Scatter(y = cbr_test_score, name = 'Test Score'), row=1, col=1)
fig.append_trace(go.Scatter(y = cbr_trn_score, name = 'Train Score'), row=2, col=1)
fig.update_layout(title = 'Train vs Test Score on CatBoostRegressor')
fig.show()

6. RandomForestRegressor¶

In [55]:
fig = make_subplots(rows = 2, cols = 1)
fig.append_trace(go.Scatter(y = rfr_test_score, name = 'Test Score'), row=1, col=1)
fig.append_trace(go.Scatter(y = rfr_trn_score, name = 'Train Score'), row=2, col=1)
fig.update_layout(title = 'Train vs Test Score on RandomForestRegressor')
fig.show()
In [56]:
X = df.drop(['Y2'],axis=1)
Y = df['Y2']
x_train, x_test, y_train, y_test = train_test_split(X,Y,test_size=0.2)
cbr = CatBoostRegressor(verbose=0, n_estimators=10000,early_stopping_rounds=100).fit(x_train,y_train)
pred = cbr.predict(x_test)
pred_trn = cbr.predict(x_train)
print(r2_score(y_train, pred_trn), r2_score(y_test, pred))

0.9936020738750924 0.9823075865850464
In [57]:
visulaize_performance_of_the_model(pred, y_test, 'CatBoost regressor')
In [59]:
df['X8'].value_counts()
Out[59]:
1    712
0     44
Name: X8, dtype: int64

Here, we're going to revert X8 values. Because the prediction of Y2(Cooling Load) causing a huge error when it tries to predict over the value of 25.¶

In [61]:
df = pd.DataFrame(pd.read_excel('C:\\Users\\harip\\INEURON_PROJECTS\\Energy Efficiency\\energy+efficiency\\ENB2012_data.xlsx'))
df
Out[61]:
X1 X2 X3 X4 X5 X6 X7 X8 Y1 Y2
0 0.98 514.5 294.0 110.25 7.0 2 0.0 0 15.55 21.33
1 0.98 514.5 294.0 110.25 7.0 3 0.0 0 15.55 21.33
2 0.98 514.5 294.0 110.25 7.0 4 0.0 0 15.55 21.33
3 0.98 514.5 294.0 110.25 7.0 5 0.0 0 15.55 21.33
4 0.90 563.5 318.5 122.50 7.0 2 0.0 0 20.84 28.28
... ... ... ... ... ... ... ... ... ... ...
763 0.64 784.0 343.0 220.50 3.5 5 0.4 5 17.88 21.40
764 0.62 808.5 367.5 220.50 3.5 2 0.4 5 16.54 16.88
765 0.62 808.5 367.5 220.50 3.5 3 0.4 5 16.44 17.11
766 0.62 808.5 367.5 220.50 3.5 4 0.4 5 16.48 16.61
767 0.62 808.5 367.5 220.50 3.5 5 0.4 5 16.64 16.03

768 rows × 10 columns

We're going to remove X1,X4 and X6. The reason is the same as we mentioned earlier in this notebook.¶

In [62]:
df.drop(['X1','X4','X6'],axis=1,inplace=True)
df
Out[62]:
X2 X3 X5 X7 X8 Y1 Y2
0 514.5 294.0 7.0 0.0 0 15.55 21.33
1 514.5 294.0 7.0 0.0 0 15.55 21.33
2 514.5 294.0 7.0 0.0 0 15.55 21.33
3 514.5 294.0 7.0 0.0 0 15.55 21.33
4 563.5 318.5 7.0 0.0 0 20.84 28.28
... ... ... ... ... ... ... ...
763 784.0 343.0 3.5 0.4 5 17.88 21.40
764 808.5 367.5 3.5 0.4 5 16.54 16.88
765 808.5 367.5 3.5 0.4 5 16.44 17.11
766 808.5 367.5 3.5 0.4 5 16.48 16.61
767 808.5 367.5 3.5 0.4 5 16.64 16.03

768 rows × 7 columns

In [63]:
X = df.drop(['Y1','Y2'],axis=1)
Y = df['Y2']
x_train, x_test, y_train, y_test = train_test_split(X,Y,test_size=0.2)
cbr = CatBoostRegressor(verbose=0, n_estimators=10000,early_stopping_rounds=100).fit(x_train,y_train)
pred = cbr.predict(x_test)
pred_trn = cbr.predict(x_train)
print(r2_score(y_train, pred_trn), r2_score(y_test, pred))

0.9763488647084976 0.9515146871956497
In [64]:
visulaize_performance_of_the_model(pred, y_test, 'CatBoost regressor')

Scenario 1 : With X8 feature and Without Y1 feature.¶

In [65]:
X = df.drop(['Y2'],axis=1)
Y = df['Y2']
x_train, x_test, y_train, y_test = train_test_split(X,Y,test_size=0.2)
cbr = CatBoostRegressor(verbose=0, n_estimators=10000,early_stopping_rounds=100).fit(x_train,y_train)
pred = cbr.predict(x_test)
pred_trn = cbr.predict(x_train)
print(r2_score(y_train, pred_trn), r2_score(y_test, pred))

0.996604989482951 0.9764696781109573
In [66]:
visulaize_performance_of_the_model(pred, y_test, 'CatBoost regressor')

Sceanrio 2 : With X8 and Y1 feature.¶

In [67]:
temp_df = df.drop(['X8'],axis=1)
temp_df
Out[67]:
X2 X3 X5 X7 Y1 Y2
0 514.5 294.0 7.0 0.0 15.55 21.33
1 514.5 294.0 7.0 0.0 15.55 21.33
2 514.5 294.0 7.0 0.0 15.55 21.33
3 514.5 294.0 7.0 0.0 15.55 21.33
4 563.5 318.5 7.0 0.0 20.84 28.28
... ... ... ... ... ... ...
763 784.0 343.0 3.5 0.4 17.88 21.40
764 808.5 367.5 3.5 0.4 16.54 16.88
765 808.5 367.5 3.5 0.4 16.44 17.11
766 808.5 367.5 3.5 0.4 16.48 16.61
767 808.5 367.5 3.5 0.4 16.64 16.03

768 rows × 6 columns

In [68]:
X = temp_df.drop(['Y2'],axis=1)
Y = temp_df['Y2']
x_train, x_test, y_train, y_test = train_test_split(X,Y,test_size=0.2)
cbr = CatBoostRegressor(verbose=0, n_estimators=10000,early_stopping_rounds=100).fit(x_train,y_train)
pred = cbr.predict(x_test)
pred_trn = cbr.predict(x_train)
print(r2_score(y_train, pred_trn), r2_score(y_test, pred))

0.995509035148715 0.9747846196154121
In [69]:
visulaize_performance_of_the_model(pred, y_test, 'CatBoost regressor')

Scenario 3 : Without X8 and with Y1 feature.¶

In [70]:
X = temp_df.drop(['Y1','Y2'],axis=1)
Y = temp_df['Y2']
x_train, x_test, y_train, y_test = train_test_split(X,Y,test_size=0.2)
cbr = CatBoostRegressor(verbose=0, n_estimators=10000,early_stopping_rounds=100).fit(x_train,y_train)
pred = cbr.predict(x_test)
pred_trn = cbr.predict(x_train)
print(r2_score(y_train, pred_trn), r2_score(y_test, pred))

0.9724107776852532 0.9669833472207664
In [71]:
visulaize_performance_of_the_model(pred, y_test, 'CatBoost regressor')

Scenario 4 : Without x8 and Y1 feature.¶

In [72]:
df = pd.DataFrame(pd.read_excel('C:\\Users\\harip\\INEURON_PROJECTS\\Energy Efficiency\\energy+efficiency\\ENB2012_data.xlsx'))
temp_df = df.drop(['X1','X6'],axis=1)
temp_df
Out[72]:
X2 X3 X4 X5 X7 X8 Y1 Y2
0 514.5 294.0 110.25 7.0 0.0 0 15.55 21.33
1 514.5 294.0 110.25 7.0 0.0 0 15.55 21.33
2 514.5 294.0 110.25 7.0 0.0 0 15.55 21.33
3 514.5 294.0 110.25 7.0 0.0 0 15.55 21.33
4 563.5 318.5 122.50 7.0 0.0 0 20.84 28.28
... ... ... ... ... ... ... ... ...
763 784.0 343.0 220.50 3.5 0.4 5 17.88 21.40
764 808.5 367.5 220.50 3.5 0.4 5 16.54 16.88
765 808.5 367.5 220.50 3.5 0.4 5 16.44 17.11
766 808.5 367.5 220.50 3.5 0.4 5 16.48 16.61
767 808.5 367.5 220.50 3.5 0.4 5 16.64 16.03

768 rows × 8 columns

In [73]:
temp_df.loc[(temp_df['X8'] > 0), 'X8']=1
temp_df
Out[73]:
X2 X3 X4 X5 X7 X8 Y1 Y2
0 514.5 294.0 110.25 7.0 0.0 0 15.55 21.33
1 514.5 294.0 110.25 7.0 0.0 0 15.55 21.33
2 514.5 294.0 110.25 7.0 0.0 0 15.55 21.33
3 514.5 294.0 110.25 7.0 0.0 0 15.55 21.33
4 563.5 318.5 122.50 7.0 0.0 0 20.84 28.28
... ... ... ... ... ... ... ... ...
763 784.0 343.0 220.50 3.5 0.4 1 17.88 21.40
764 808.5 367.5 220.50 3.5 0.4 1 16.54 16.88
765 808.5 367.5 220.50 3.5 0.4 1 16.44 17.11
766 808.5 367.5 220.50 3.5 0.4 1 16.48 16.61
767 808.5 367.5 220.50 3.5 0.4 1 16.64 16.03

768 rows × 8 columns

In [74]:
X = temp_df.drop(['Y2'],axis=1)
Y = temp_df['Y2']
x_train, x_test, y_train, y_test = train_test_split(X,Y,test_size=0.2)
cbr = CatBoostRegressor(verbose=0, n_estimators=10000,early_stopping_rounds=100).fit(x_train,y_train)
pred = cbr.predict(x_test)
pred_trn = cbr.predict(x_train)
print(r2_score(y_train, pred_trn), r2_score(y_test, pred))

0.9936908558985426 0.9788257878445978
In [75]:
visulaize_performance_of_the_model(pred, y_test, 'CatBoost regressor')

Scenario 5 : With modified X8 and Y1 feature¶

From the above 5 scenario's, we can pic 4th . The reason is, only few points away from center line when compared to other 3 scenario's. Without X8 and with Y1 feature, the Y2 predictions are good.¶

We can check this for Y1 prediction also. We have to know whether X8 is considered or not.¶

In [76]:
X = temp_df.drop(['Y1','Y2'],axis=1)
Y = temp_df['Y1']
x_train, x_test, y_train, y_test = train_test_split(X,Y,test_size=0.2)
cbr = CatBoostRegressor(verbose=0, n_estimators=10000,early_stopping_rounds=100).fit(x_train,y_train)
pred = cbr.predict(x_test)
pred_trn = cbr.predict(x_train)
print(r2_score(y_train, pred_trn), r2_score(y_test, pred))

0.9980610932677114 0.9974413562218192
In [77]:
visulaize_performance_of_the_model(pred, y_test, 'CatBoost regressor')

Without X8, Y1 prediction is also good. But it is bit important for X7 feature. having 0.2 coorelation with each other. So, we can include X8 for Y1 Prediction. So, we have to create two seperate models for better predictions.¶

In our project development, we're going to follow the below rules :¶

1 .Prediction of Y1 with the independent variables of X2, X3,X5,X6,X7 and X8.¶

2 .Prediction of Y2 with the independent variables of X2, X3,X5,X6,X7,X8 and dependent feature of Y1. The reason for adding Y1 for Y2 prediction is¶